Methods and systems for tuning a static model

ABSTRACT

Methods and systems are provided for tuning a static model with multiple operating points to adjust model performance without retraining the model or triggering a new regulatory clearance. In one embodiment, a method comprises, responsive to a request to tune a model, obtaining a tuning dataset including a set of medical images, executing the model using the set of medical images as input to generate model tuning output, and determining, for each operating point of a set of operating points, a set of tuning metric values based on the tuning dataset and the model tuning output relative to each operating point. An operating point from the set of operating points may be selected based on each set of tuning metric values and, upon a request to analyze a subsequent medical image, a representation of a finding output from the static model executed at the selected operating point.

FIELD

Embodiments of the subject matter disclosed herein relate to a methodfor tuning a static model.

BACKGROUND

Radiological medical imaging systems are often used to monitor, image,and diagnose a subject. To increase the efficacy of such systems, theuse of artificial intelligence models to automatically identify andcharacterize radiological images is becoming more widespread.

BRIEF DESCRIPTION

In one embodiment, a method, responsive to a request to tune a staticmodel, comprises: obtaining a tuning dataset including a set of medicalimages; executing the diagnostic model using the set of medical imagesas input to generate a model output; determining, for each operatingpoint of a set of operating points, a set of tuning metric values basedon the tuning dataset and the model output relative to each operatingpoint; selecting an operating point from the set of operating pointsbased on each set of tuning metric values; and, upon a request toanalyze a subsequent medical image, outputting a representation of anoutput of the static model executed at the selected operating point.

It should be understood that the brief description above is provided tointroduce in simplified form a selection of concepts that are furtherdescribed in the detailed description. It is not meant to identify keyor essential features of the claimed subject matter, the scope of whichis defined uniquely by the claims that follow the detailed description.Furthermore, the claimed subject matter is not limited toimplementations that solve any disadvantages noted above or in any partof this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be better understood from reading thefollowing description of non-limiting embodiments, with reference to theattached drawings, wherein below:

FIG. 1 shows an example x-ray imaging system according to an embodiment;

FIG. 2 is a flow chart of a method for tuning a static model;

FIGS. 3A and 3B show non-limiting examples of data matrices that may begenerated using the method presented in FIG. 2;

FIG. 4 shows a non-limiting example of graphical data that may be outputusing the method presented in FIG. 2;

FIGS. 5A and 5B show a non-limiting example graphical user interfacethat may allow medical staff to tune a static model in a clinicalsetting;

FIG. 6 shows a non-limiting example graphical user interface that mayallow medical staff to choose a model operating point for radiologicalimage analysis in a clinical setting; and

FIGS. 7A and 7B show a non-limiting example of an annotated clinicalimage that may be part of a tuning dataset used to tune a static modelvia the method presented in FIG. 2.

DETAILED DESCRIPTION

Clinical development has undergone a transformation in recent years dueto the convergence of digital data sources and the efficient use ofartificial intelligence (AI) as well as machine-learning models toidentify clinically meaningful patterns in data. Al programs have beendeveloped and applied to practices such as diagnosis processes, protocoldevelopment, drug development, personalized medicine, and patientmonitoring/care. For example, AI can learn features from a large volumeof healthcare data, and then use the obtained insights to assistclinical practices in treatment design or risk assessment.

More specifically, AI models have demonstrated remarkable progress inradiological image recognition tasks. Historically, radiological medicalimages have been visually assessed by trained medical professionals forthe detection, characterization, and monitoring of diseases. However, asthere is a desire for greater efficacy and efficiency in clinical careand AI methods excel at automatically recognizing complex patterns inimaging data, the use of AI in the identification and characterizationof radiographic characteristics is becoming increasingly prevalent. Forexample, lung cancer screening can help identify pulmonary nodules, withearly detection being lifesaving in many patients. AI is now routinelyused in the automatic identification of these nodules and the subsequentcharacterization of them as benign or malignant. Similarly, AI has beenused to identify and characterize lesions during abdominal/pelvicimaging, colonic polyps from colonoscopy imaging, microcalcificationsfrom mammography imaging, tumors from brain imaging, and so on. As such,a seamlessly integrated AI component within the imaging workflow mayincrease efficiency, reduce errors, and achieve objectives with minimalmanual input by providing trained radiologists with pre-screened imagesand identified features. Thus, by automating such processes,institutions may decrease time until diagnosis as well as costs andstaffing needs typically associated with such tasks.

Commercially deploying clinical AI models into a medical settingrequires a very high burden of proof to regulatory bodies for safety andeffectiveness clearance. Therefore, AI developers are limited in howoften they can retrain and deploy new models, as all modifications mustbe reviewed and cleared by a regulatory body. Thus, the same approvedstatic model may be applied across a wide range of clinical settings(e.g., critical care units, emergency rooms, primary care). However, itis anticipated that the optimal operating point for the model may varybetween settings/institutions due to patient population differences,imaging equipment variation, radiologist practice/preferences, differingsensitivity versus specificity needs, etc., creating a challenge duringcommercial deployment as not all customers may be pleased withperformance based on these distinctions. As such, some static models mayinclude more than one operating point, or threshold, to allow customersto optimize the model by selecting the operating point that best suitstheir particular needs. For example, in the case of chest x-rays forcertain findings (i.e. life threatening), some radiologists maydetermine all suspicious findings as a positive needing follow-up forpatients and, thus, opt to use an operating point with high sensitivitywhen deploying a static model. Alternatively, other radiologists willcall an image positive only if the radiological finding isobvious/evident since a subtle finding may be a false positive and thecondition should also have suffering vital signs to require treatmentand, thus, may choose to use an operating point with a high level ofspecificity when deploying a model for x-ray characterization. However,there are currently no methods to help a customer choose which of themodel's operating points is optimal for their needs other than usingtrial and error or in-depth statistical analysis.

Thus, according to embodiments disclosed herein, a method may bedeployed to tune a model's operating point to enable desired performancewithout retraining the model or potentially triggering a new regulatoryclearance. For example, a method may be employed to calculate one ormore performance metrics of a model for maximum accuracy based oninstitution preferences/thresholds for image classification (e.g., inthe case of testing for lung nodules, an institution may choose toclassify all suspicious chest x-ray findings as positive). Once theoperation point for maximum accuracy has been determined, the model'soperating threshold may be adjusted to best suit/serve the needs aninstitution's clinical practice.

FIG.1 depicts an x-ray imaging system that may be used to capture x-rayimages that may be employed to tune a static model using the methodpresented in FIG. 2. FIGS. 3A and 3B show example data matrices ofoperating point threshold variation that may be generated using themethod presented in FIG. 2. FIG. 4 shows an example of graphical datathat may be output using the method presented in FIG. 2. FIGS. 5A and 5Bshow an example graphical user interface (GUI) that may allow medicalstaff to tune a static model in a clinical setting. FIG. 6 shows anexample GUI that may allow medical staff to choose an optimal modeloperating point for radiological image analysis in a clinical setting.FIGS. 7 and 8 show examples of annotated clinical images that may bepart of a tuning dataset used to tune a static model via the methodpresented in FIG. 2.

Turning now to FIG. 1, a block diagram of an x-ray imaging system 100 inaccordance with an embodiment is shown. The x-ray imaging system 100includes an x-ray source 111 which radiates x-rays, a stand 132 uponwhich the subject 115 stands during an examination, and an x-raydetector 134 for detecting x-rays radiated by the x-ray source 111 andattenuated by the subject 115. The x-ray detector 134 may comprise, asnon-limiting examples, a scintillator, one or more ion chamber(s), alight detector array, an x-ray exposure monitor, an electric substrate,and so on. The x-ray detector 134 is mounted on a stand 138 and isconfigured so as to be vertically moveable according to an imaged regionof the subject.

The operation console 180 comprises a processor 181, a memory 182, auser interface 183, a motor drive 185 for controlling one or more motors143, an x-ray power unit 186, an x-ray controller 187, a camera dataacquisition unit 190, an x-ray data acquisition unit 191, and an imageprocessor 192. X-ray image data transmitted from the x-ray detector 134is received by the x-ray data acquisition unit 191. The collected x-rayimage data are image-processed by the image processor 192. A displaydevice 195 communicatively coupled to the operating console 180 displaysan image-processed x-ray image thereon.

The x-ray source 111 is supported by a support post 141 which may bemounted to a ceiling (e.g., as depicted) or mounted on a moveable standfor positioning within an imaging room. The x-ray source 111 isvertically moveable relative to the subject or patient 115. For example,one of the one or more motors 143 may be integrated into the supportpost 141 and may be configured to adjust a vertical position of thex-ray source 111 by increasing or decreasing the distance of the x-raysource 111 from the ceiling or floor, for example. To that end, themotor drive 185 of the operation console 180 may be communicativelycoupled to the one or more motors 143 and configured to control the oneor more motors 143.

The x-ray power unit 184 and the x-ray controller 182 supply power of asuitable voltage current to the x-ray source 111. A collimator (notshown) may be fixed to the x-ray source 111 for designating anirradiated field-of-view of an x-ray beam. The x-ray beam radiated fromthe x-ray source 111 is applied onto the subject via the collimator.

A camera 120 may be positioned adjacent to the x-ray source 111 and maybe co-calibrated with the x-ray source 111. The x-ray source 111 and thecamera 120 may pivot or rotate relative to the support post 141 in anangular direction 119 to image different portions of the subject 115.The camera 120 may comprise an optical camera that detectselectromagnetic radiation in the optical range. Additionally oralternatively, the camera 120 may comprise a depth camera or rangeimaging camera. As an illustrative and non-limiting example, the camera120 configured as a depth camera may include an optical camera, aninfrared camera, and an infrared projector which projects infrared dotsin the field-of-view of the camera 120. The infrared camera images thedots, which in turn may be used to measure depth within the opticalcamera of the camera 120. As another illustrative and non-limitingexample, the camera 120 may comprise a time-of-flight camera. The camera120 is communicatively coupled to the camera data acquisition unit 190of the operation console 180. Camera data acquired or generated by thecamera 120 may thus be transmitted to the camera data acquisition unit190, which in turn provides acquired camera image data to the imageprocessor 192 for image processing. For example, as described furtherherein, the image processor 192 may process the acquired camera imagesto identify a position of a desired anatomical region for imaging and/orto measure or estimate the thickness of the subject 115 at the desiredanatomical region. In some examples, console 180 and/or PACS 196 mayinclude a report module configured to identify and annotate radiologicalfindings in acquired x-ray images (e.g. based on the radiology reportusing natural language processing (NLP)). Image processor 192 may sendprocessed images to an edge device 197 and/or a picture archiving andcommunication system (PACS) 196 to which image processor 192 iscommunicatively coupled. Edge device 197 may be an edge processingdevice, a cloud processing device, or an extra computing device coupledto a network 198. Further, network 198 may be communicatively coupledwith PACS 196 so image data may be transferred between network 198, PACS196, and/or edge device 197.

Images captured using x-ray imaging system 100 may be subsequently usedto tune a static model deployed in automated radiological imagerecognition tasks. For example, a static model may identify andcharacterize various radiological findings, such as lesions,microcalcifications, or tumors in x-ray images acquired using aradiological imaging system (such as imaging system 100) and output apositive or negative indication for the finding based on theidentification and/or characterization of the findings. The static modelimplemented in radiological image characterization may output a score(e.g., ranging from 0-100), with a higher score (e.g., closer to 100)more likely indicating a positive finding (e.g., the presence ofdisease) and a lower score (e.g., closer to 0) more likely indicating anegative finding (e.g., no disease). In some cases, different modelthresholds or operating points may be used by the static model in thegeneration of these scores. For example, the static model may have fiveoperating points corresponding to thresholds of 50%, 70%, 80%, 90%, and100%, with each threshold impacting a parameter (e.g., sensitivity,specificity) of the model's performance and, thus, impacting the scoregenerated by the model; thereby changing the amount of false and truepositives. As such, the model may remain static (e.g., not need to bere-trained and undergo new regulatory clearance) but may be optimizedfor performance based on the clinical setting/needs of the customer.

Currently, in order for customers to determine which operating point isideal for their institution, methods involving deep statistical analysisor trial and error may be employed. These tuning methods may be time-and/or resource-intensive, and/or may not result in the model beingtuned as optimally as possible. As such, according to the embodimentsdisclosed herein, a method is provided that may be employed by customersto easily identify which operating point of a static model may best suittheir needs based on tuning the model using their own clinical practicedata. For example, the method may tune a static model using an imagedataset selected from imaging data specific to a given clinical setting(e.g., an institution, a department, unit, or ward, etc.) to determinean optimal operating point for the static model, where the static modelis used in the identification and characterization of image findings,patient positioning, and/or proper protocol selection (e.g., a chestprotocol is used for imaging a chest). In this way, the operating pointof the static model may then be adjusted to optimize performance in theclinical setting.

The systems and methods are described herein with respect to an x-rayimaging system but the methods disclosed herein may be implemented invirtually any other imaging environment without departing from the scopeof this disclosure. For example, the methods disclosed herein may beapplied to tune a model used to identify findings in images captured viaan ultrasound system, magnetic resonance imaging (MRI), computerizedtomography (CT) scans, positron emission tomography (PET) scans, singlephoton emission computed tomography (SPECT) scans, and/or visible lightcameras. Further, while the static model is described herein as beingstored on and tuned on a specific imaging device (e.g., an x-raymachine), in some examples, the model may be stored and tuned on one ormore other devices within an imaging system, such as a PACS (e.g., PACS196 of x-ray imaging system 100) or another suitable computing device(e.g., edge device 197) communicatively coupled to the imaging system.

FIG. 2 is a flow chart of a method 200 for tuning a static model withmultiple operating points to optimize performance within a clinicalsetting. Method 200 may be executed using computer readable instructionsstored in the non-transitory memory of a computing device of an x-rayimaging system or another imaging modality/system located at aninstitution (e.g., hospital, imaging unit, ward, department), such asmemory 182 of FIG. 1. In other examples, method 200 may be executed byanother computing device without departing from the scope of thisdisclosure, such as a PACS (e.g., PACS 196 of FIG. 1) or an edge device(e.g., edge device 197 of FIG. 1).

At 202, a default model operating point for a static model may beselected. The static model may be trained to identify and/orcharacterize one or more suitable image parameters of clinical images,such as diagnostic findings (e.g., the presence of absence of lungnodules), patient positioning, exposure, image noise/artifacts, properprotocol selection, etc. The default model operating point may be acommercially set operating point or a default operating point previouslydetermined via tuning according to a previous iteration of method 200.Some static models may have only a few predefined possible operatingpoints (e.g., 50, 60, 70, 80, 90) but the selected operating point maybe different than the predefined operating points (e.g., the tuningprocess described herein may be carried out with 100 different operatingpoints). Thus, in some examples, the predefined operating point closestto the operating point selected after tuning may be selected (e.g., ifthe tuning process identifies 59 as the optimal operating point, theselected operating point may be the closest predefined operating point,which in the prior example may be 60). The tuning process may be carriedout with only the possible predefined operating points (e.g., 50, 60,70, 80, 90) or the tuning process may be carried out with a range ofadditional possible operating points (e.g., 50-100). In one example, themodel's operating points may be different thresholds against whichoutput of the model may be compared to determine whether or not acertain image parameter is present in an image, and which affect one ormultiple tuning metrics such as sensitivity, specificity, accuracy,positive predictive value (PPV), and/or negative predictive value (NPV).A user such as medical staff may select the model's default operatingpoint using a GUI such as the GUI presented in FIG. 6.

At 204, method 200 includes determining if a request to tune the staticmodel has been received. A user, such as a clinician or other medicalstaff, may input a request for the model to be tuned using a suitableuser input device, e.g., touch input to a graphical user interface(GUI), such as the GUI presented in, and further described with respectto, FIGS. 5A and 5B. If a request to tune the model has not beenreceived, method 200 proceeds to 206, where the model may be executedwith the selected default operating point on subsequent clinical images,after which method 200 may return to the start. The subsequent clinicalimages may be x-ray images obtained by the x-ray imaging system, and themodel may be executed to generate output that is compared to the defaultoperating point in order to provide an indication of the parameter themodel is trained to identify/characterize.

If a request to tune the model has been received, method 200 proceeds to208 to receive a selection of a tuning metric. The tuning metric hereinmay be defined as an umbrella of evaluation criteria by which operatingpoints may be distinguished based on findings from binary output. Tuningmetrics may include sensitivity, specificity, accuracy, balancedaccuracy, positive predictive value (PPV), negative predictive value(NPV), and/or the number of false positives/false negatives/truepositives/true negatives per day in the institution. The user mayrequest the model be tuned based on clinical setting, the specificpatient/patient population, imaging equipment variation, and/orradiologist practice/preferences. For example, the default operatingpoint selected at 202 may be set to identify and characterizeradiological findings with a high degree of specificity to decreasepotential false alarms during automated identification (e.g., theidentification of plural effusions from chest x-rays). While a highdegree of specificity may be advantageous in certain clinical settingssuch as in an intensive care unit (ICU) in which patients arecontinuously monitored, in other settings staff may prefer the model tooperate with a higher degree of sensitivity than specificity (e.g., insettings where the patient is released within the same day as beingadmitted). For example, a higher degree of sensitivity may be moreadvantageous than specificity when the model is deployed in emergencyrooms (ERs) as potential mischaracterization of a radiological artifactmay be life-threatening or contagious. As a non-limiting example, apatient may enter a clinic with tuberculosis. If a model is deployedwith an operating point having high degree of specificity, the model maynot identify tuberculosis in the chest x-ray of said patient and thepatient may be released. In contrast, the same model using an operatingpoint tuned for sensitivity may identify the same chest x-ray aspositive for tuberculosis.

At 210, an annotated tuning dataset may be obtained. The annotatedtuning dataset may be representative of the institution and may includeimage data (e.g., x-ray images) annotated by one or more experts at theinstitution and/or according to institutional preferences to set themodel's performance to align with preferences of the institution and/ora sub-group within the institution (e.g., department, ward, etc.). Forexample, the tuning dataset may be annotated by a selected radiologistwho has been trained to indicate an image parameter (e.g., a specificdiagnostic finding) according to institutional regulations andpreferences, which in some examples may include ward or unit-basedpreferences (e.g., annotated for a high specificity threshold forpatients within an ICU, annotated for a high sensitivity threshold forpatients within the ER, annotated for a high sensitivity in a patientpopulation particularly susceptible to a certain disease). In someexamples, the tuning dataset may be representative of a specific patientpopulation that is likely to be imaged at the institution, due at leastin part to the images in the tuning dataset being obtained at theinstitution, obtained at the same geographical region of theinstitution, obtained of patients having the same demographic make-up asthose typically admitted to/imaged by the institution, etc. For example,if the static model is deployed on a device of a pediatric hospital, theimages in the tuning dataset may be images of children. The annotatedtuning dataset may be obtained automatically or manually. Automaticcollection of image data representative of the clinical practice may beimplemented using a Digital Imaging and Communication in Medicine(DICOM) push or pull data interchange protocol in which images are sentto a specified destination on an edge server (e.g., edge device 197 ofFIG. 1) or cloud location (e.g., network 198 of FIG. 1). The image datamay then be selected from a specified destination and manually annotatedby one or more users, or automatically annotated. Automatic annotationmay use NLP on radiology reports to annotate the selected image data, atleast in some examples.

In some examples, representative image data for the tuning dataset maybe manually selected from clinical images stored on the network or PACScommunicatively coupled to the imaging system (e.g., network 198 or PACS196 of imaging system 100 of FIG. 1) by an expert (e.g., radiologist)who subsequently manually annotates the selected images. In anotherexample, manual image annotation may be implemented by displaying themodel's results (explained in more detail below) to a user on a userinterface of the imaging system (e.g., user interface 183 of imagingsystem 100 of FIG. 1) or a web application communicatively coupled tothe network or PACS linked to the imaging system. The user may thenannotate the image with a finding of being in agreement with thedisplayed result or a finding of being in disagreement with thedisplayed result. For example, a displayed clinical image may belabelled with the model's result and include a hand icon with its thumbup as well as a second hand icon with its thumb down. If the user doesnot agree with the model's characterization/result of the imageparameter(s) (e.g., radiological finding) presented, the user may selectthe icon with the thumb down thereby annotating the image.Alternatively, if the user does agree with the model's result, the usermay select the icon with the thumb up to annotate the image as being acorrect finding.

In an embodiment, the tuning dataset may be obtained when tuning isrequested at 204. In another embodiment, the tuning dataset may beobtained in advance and stored in memory of the computing device, andthen reused when tuning is requested at 204. In some examples, thetuning dataset may be continuously collected during clinical usage.

At 212, the annotated images may be entered into the model and the modelexecuted to generate model output. The model output may include, foreach image of the tuning dataset that entered into the model, a valuethat reflects a likelihood that the image has the image parameter(s) themodel is trained to identify. For example, if the model is trained todetermine if a finding of lung nodules is present in images, the modeloutput may include a value (e.g., from 0-100, 0-10, etc.) that indicatesa likelihood that the image includes a finding of lung nodules, withhigher values indicating a higher likelihood.

At 214, a matrix is populated with tuning metric value(s) for each imagebased on the model output relative to a first operating point. The firstoperating point may be the default operating point selected at 202, arandomly selected operating point (e.g., randomly selected from the setof operating points discussed above), a lowest or highest valueoperating point of the set of operating points, or the first operatingpoint may be an operating point selected by the user (e.g., if the modelhas five operating points, the user may select any of these fiveoperating points).

In one embodiment, the matrix may be populated with values determined bydirectly comparing the model output to the annotation using a binarysystem. True positive (TP) and true negative (TN) findings within thetuning dataset (e.g., a finding of lung nodules when lung nodules arepresent, a finding of no lung nodules when no lung nodules are present)and the model output may be labelled as 0 whereas false positive (FP)and false negative (FN) findings within the tuning dataset and the modeloutput (e.g., a finding of lung nodules when no lung nodules arepresent, a finding of no lung nodules when lung nodules are present) maybe labelled as 1. In some embodiments, the values of for the TPs/TNs andFPs/FNs may be reversed (e.g., TPs and TNs may be labelled as 1, FPs andFNs may be labelled as 0). The matrix may be comprised of rowscorresponding to the number of images in the tuning dataset and columnscorresponding to the different operating points of the model (see theexample matrix presented in FIG. 3A for further detail). In someexamples, such as when the tuning metric is maximum accuracy, the matrixmay be populated with a tuning metric error value for each image at thefirst operating point by determining the absolute value of the modeloutput finding (labelled as a 1 or 0) minus the annotation finding(labelled as a 1 or 0). For example, if both the model and the expertdetermine an image as being a TP or TN for a finding, a value of 0 willbe entered into the matrix as |1-1|=0. Alternatively, if the modeloutput is different than the annotation (e.g., a FP or a FN), a value of1 will be generated within the matrix as |1-0|=1 and |0-1|=1.

In some examples, the matrix may be populated with multiple values foreach image, such as a value (or character) indicating if the image was aTP, a TN, a FP, or a FN. The number of TPs, TNs, FPs, and FNs for eachoperating point in a populated matrix may then be used in conjunctionwith the number of patients assessed for a disease per day and anoccurrence rate of the disease in the given patient population todetermine other metrics such as the NPV, PPV, sensitivity, specificity,accuracy, balanced accuracy, the Youden index (e.g., the sum of thesensitivity plus the specificity minus one), and/or the number of falsepositives/false negatives/true positives/true negatives per day in theinstitution. For example, the number of true positives per day in theinstitution may be determined by multiplying the number of patientsassessed for disease per day by the prevalence of disease and thesensitivity. In another example, the number of false positives per dayin the institution may be equal to: the number of patients assessed fordisease per day*(1−the prevalence of disease)*(1−specificity). Inanother example, the number of false negatives per day in theinstitution may be equal to: the number of patients assessed for diseaseper day*(the prevalence of disease)*(1−sensitivity). These additionalmetrics may provide additional help to the user when evaluating theimpact of the given operating point to the institution. An examplematrix summarizing TPs, TNs, FPs, and FNs for a plurality of images at aplurality of different operating points is shown in FIG. 3B andexplained in more detail below.

At 216, for each additional operating point of the set of operatingpoints, the matrix is populated with respective tuning metric values foreach image based on the model output relative to each respectiveoperating point. For example, the model output for each image may becompared to a second operating point to determine a positive or negativefinding, and the tuning metric value for each image for the secondoperating point may be determined by comparing the positive or negativefinding to the finding of the corresponding annotated image, similar tothe determination of the tuning metric values for the first operatingpoint.

At 218, a target model operating point may be determined based on thepopulated matrix and the selected tuning metric. For example, if theselected tuning metric is maximum accuracy, the tuning metric values ineach respective column may be summed. As each column representsperformance at one operating point, the column with the lowest sumrepresents the target operating point for the tuning dataset as it hasthe least amount of error compared to the expert annotations. Ifmultiple operating points share the same minimum error (e.g., the sametotal column sum), various methods may be implemented to furtherdifferentiate which operating point may be optimal for the customer(e.g., the median operating point may be selected, or the multipleoperating points may be tuned for a second parameter).

The target operating point may be selected according to differentmethods based on which tuning metric is selected. In one embodiment, ametric curve may be determined based on the populated matrix, and atarget operating point may be selected using the metric curve and theselected tuning metric, such as sensitivity, specificity, accuracy, PPV,and/or NPV (see FIG. 4 for example metric curves). The target operatingpoint may be selected based on maximum accuracy, maximum Youden index,curve proximity to the upper left corner, etc. Alternatively, a user maybe provided with graphs that represent performance metrics along themetric curve or data summations generated from metric curves and theuser may select a target operating point that tailors the model'sperformance to their desired metric. Further, in addition to adaptingthe operating point to a specific clinical setting/user/users, method200 may also recalibrate the p value based on the acquired data—eitherto reflect probability or to reflect rank. The p value may berecalibrated via Platt scaling, a calibration tree, isotonic regression,or another suitable method.

At 220, method 200 optionally includes adjusting the operating point ofthe model. For example, the target model operating point identified at218 may be presented to a user via a user interface and the user mayselect to adjust the operating point to the target operating point, orthe user may choose not to adjust the operating point and maintain thedefault model operating point selected at 202. The selected operatingpoint may then be saved in the memory of the computing device of theimaging system or memory of the computing device communicatively coupledto the imaging system. In some examples, the operating point may beautomatically adjusted and saved if the target operating point isdifferent than the default operating point. At 222, the model may beexecuted on subsequent clinical images (e.g., x-ray images) using thedetermined target operating point, when indicated (e.g., in response toa user request to execute the model and/or in response to reception of aclinical image that is to be entered as input to the model). Forexample, a clinical image may be entered into the model as input, themodel may output a value indicating a likelihood that the clinical imagehas a specific image parameter (such as a finding of lung nodules), andthe model output may be compared to the target operating point todetermine if the clinical image has the image parameter. As anon-limiting example, the model may be trained to detect lung nodules,and may output a likelihood value of 8 when a first clinical image isinput into the model. If the target operating point is 7, the firstclinical image may be determined to have a positive finding of lungnodules. If the target operating point is 9, the first clinical imagemay be determined to have a negative finding of lung nodules. In thisway, the interpretation of the model output may be adjusted based ontuned operating point, which may affect whether or not specificparameters are identified, without adjusting the static model itself.Method 200 may then return to the start.

FIG. 3A shows a non-limiting example of a matrix 300 populated with aplurality of tuning metric error values that may be generated usingmethod 200. As previously described, each row of the matrix 300corresponds to one image of an annotated tuning dataset with the numberof rows determined by the number of images comprising the tuning dataset(e.g., a tuning dataset comprised of 100 images would generate a matrixwith 100 rows, a tuning dataset comprised of 200 images would generate amatrix with 200 rows). Each column of the matrix corresponds to adifferent operating point (e.g., as shown, the operating points arethresholds). Matrix 300 is comprised of 100 columns and was thusgenerated using a static model that has 100 different operating points.In other examples, a matrix generated using method 200 may have as fewas two columns (e.g., generated from a model that has two operatingpoints) but less than 100 columns or may have more than 100 columns.Populated data within the matrix (shown as 0s and 1s) correspond towhether the model output matches a corresponding annotation for eachimage in an annotated tuning dataset. A designation of 1 indicates themodel output did not match the corresponding annotation as a positive ornegative finding whereas a designation of 0 indicates the model outputmatches the corresponding annotation as a positive or negative finding.Thus, by summing each column, an overall tuning metric error for eachoperating point may be determined. The operating point representing thecolumn with the lowest sum may then be determined as an optimaloperating point for maximum accuracy, as the model output relative tothat operating point most closely matches the annotations within thetuning dataset. For example, the sum 302 of column 53 in matrix 300 isten, which is the lowest overall error of the matrix. Thus, out ofcomparing 1 to N images of a tuning dataset with the model output,output from operating point 53 did not match ten image annotations. Bycomparison, operating points one, two, and four did not match theannotations of 99 images within the tuning dataset. Thus, operatingpoint 53 (e.g., a threshold of 53) may be determined as the optimaloperating point for the given tuning dataset.

FIG. 3B shows another example matrix 350 which may be generatedaccording to method 200. Matrix 350 may be populated based on comparingmodel output for a plurality of images to a plurality of operatingpoints to generate model findings, and then comparing the model findingsto expert findings for each image of the plurality of images. Incontrast to matrix 300, matrix 350 may represent a sum of a largermatrix or plurality of matrices. Matrix 350 includes a plurality ofcolumns, with each column representing an operating point. As shown inFIG. 3B, matrix 350 includes columns for operating points (e.g.,thresholds) of 10, 20, 30, 40, 80, 90, and 99, though other operatingpoints are within the scope of this disclosure. For visual purposes,select operating points (e.g., 50, 60, and 70) have been left off ofmatrix 350.

For each operating point, matrix 350 includes a summation of a pluralityof tuning metric values, where each summation indicates how many imagesfrom a tuning dataset were determined to have that tuning metric value.For example, matrix 350 includes, for each operating point, the numberof images determined to be true positives, false positives, truenegatives, and false negatives, as described above with respect to FIG.2. Based on the tuning metric values for each operating point, variousoverall tuning metrics may be calculated. As shown, the overall tuningmetrics calculated from matrix 350 include sensitivity, specificity,PPV, NPV, accuracy, and balanced accuracy.

Using a static model trained to detect lung nodules as an example, ifthe static model is tuned to operate with a relatively low operatingpoint (e.g., an operating point of 10), nearly all instances of lungnodules may be detected (e.g., a sensitivity of 97.8%). However, thislow operating point may result in a relatively high number of imagesthat do not have lung nodules being classified as having lung nodules(e.g., a specificity of 58.1%). By increasing the operating point, thenumber of false positives may be reduced (e.g., an operating point of 90may result in only 17 false positives, compared to 104 false positivesfor an operating point of 10), but correspondingly the number of falsenegatives may increase (e.g., from 4 to 44). Thus, the user may selectwhich operating point provides the best balance of sensitivity,specificity, accuracy, etc., for the needs of the specificinstitution/department.

FIG. 4 shows non-limiting examples of metric curves 400 that may begenerated using method 200 to calculate different performance metrics ofmodel operating points. For example, a first metric curve 402illustrates thresholds for sensitivity and specificity based on datafrom a populated matrix with tuning metric error for each operatingpoint of the model (e.g., matrix 300 of FIG. 3A). A second set of metriccurves 404 generated from a populated matrix illustrates thresholds forsensitivity, specificity, accuracy, and balanced accuracy. A third setof metric curves 406 generated from a populated matrix illustratesthresholds for PPV and NPV. A fourth set of metric curves 408 generatedfrom a populated matrix illustrates thresholds for false negatives, truepositives, and false positives. As an optimal operating point may be acompromise between conflicting needs (e.g., sensitivity andspecificity), multiple performance metrics may be simultaneouslyassessed before selecting an operating point. Thus, in one example, apotential user interface (e.g., a GUI such as the GUI presented in FIG.6) may show all the graphs presented in FIG. 4 to a user at the sametime (e.g., all four graphs may be presented on a single screen) so thatthe user may consider all implications of using a selected operatingpoint.

In another example, the user may use the data presented in the secondset of metric curves 404, the third set of metric curves 406, and thefourth set of metric curves 408 as a basis for selecting an optimaloperating point. A currently selected operating point (e.g., 0.70) isshown on each set of metric curves, which may enable the user toevaluate all the different metrics for the selected operating point andconfirm that the metrics are acceptable. For example, the point on themetric curve 402 may represent the currently-selected operating point,and the dashed line on each of the remaining sets of metric curves mayrepresent the same, currently selected operating point. The user mayadjust the operating point by selecting different points on the metriccurve 402, or via another form of user input, as explained below. Insome examples, the user may select an operating point threshold directlyfrom a displayed graph via input through a mouse (e.g., by clicking onthe graph) communicatively coupled to a user interface. In someexamples, the user interface may be a touchscreen and the user may toucha threshold on a graph to select a different operating point. In someexamples, the user may adjust the threshold up or down using arrow keyson a keyboard communicatively coupled to the user interface. In someexamples, an operating point may be automatically selected based on userdefined criteria (e.g., the Youden index, maximum balanced accuracy),with the selection appearing on the graphical output so that the usermay confirm the selected operating point before use in image analysis.

Metric curve 402 may be generated by plotting a true positive rate(TPR), referred to herein as sensitivity, against a false positive rate(FPR), referred to herein as one minus specificity (e.g.,FPR=1−specificity) for each tested operating point. The sensitivity isthe ratio of correctly identified positives among all actual positives(e.g., the percentage of actual positive images that were correctlyindicated by the model as having a positive finding), while thespecificity may be defined as the actual negatives that are correctlyidentified as negative (e.g., the percentage of actual negative imagesthat were correctly indicated by the model as having a negativefinding). Metric curve 402 may be used to automatically select anoptimal operating point as previously described. Alternatively,generated metric curves may be output to a GUI (such as the GUIpresented in FIG. 6) on a display device so that a user may select anoptimal operating point based on the data provided. For instance, a userin a primary care setting may want the model's performance to bebalanced between specificity and sensitivity. Thus, if first metriccurve 402 was displayed, the user might select an operating point in themiddle of the curve for optimal performance (e.g., the operating pointrepresented by the dot on the curve of metric curve 402). Alternatively,if the user works in an ER, he/she may want to select an operating pointwith a higher level of sensitivity (e.g., the operating pointrepresented by the X on the curve of metric curve 402) as falsenegatives may be life threatening to patients. On the other hand, a userin an ICU may prefer to minimize false positives as the patients arebeing continuously monitored and thus the user may select an operatingpoint with high specificity (e.g., the operating point represented bythe square on the curve of metric curve 402) for optimal performance.

In the second set of metric curves 404, the third set of metric curves406, and the fourth set of metric curves 408, the x-axis may representthe operating point (e.g., threshold) of the model yielding the metricvalue defined by the y-axis. The vertical dashed line within the secondset of metric curves 404, the third set of metric curves 406, and thefourth set of metric curves 408 may represent the model's currentoperating point (e.g., without tuning). The model's current operatingpoint in the second set of metric curves 404, the third set of metriccurves 406, and the fourth set of metric curves 408 may be at athreshold of 0.7. For the second set of metric curves 404 and the thirdset of metric curves 406, the y-axis may represent the determined tuningmetric error of each displayed metric normalized to one. Values close toor at zero on the y-axis may correspond to a higher degree of tuningmetric error whereas values close to or at one may correspond to a lowerdegree of or no (e.g., a value of 1) tuning metric error. The y-axis ofthe fourth set of metric curves 408 may represent the total number ofoccurrences for each displayed tuning metric (e.g., the total number offalse negatives per day).

In some examples, the user may utilize a metric curve to deploy themodel with an operating point based on multiple calculated metrics suchas sensitivity, specificity, accuracy, and balanced accuracy (e.g., thearithmetic mean of the true positive rate and true negative rate). Thus,if second set of metric curves 404 was displayed, the user may select anoperating point based the tuning metric error of a specificity curve414, a sensitivity curve 410, an accuracy curve 416, and a balancedaccuracy curve 412 such as the operating point indicated by the X whichcorrelates to a threshold of 0.4. The 0.4 threshold has a higher degreeof sensitivity and balanced accuracy as compared to the model's currentoperating point threshold of 0.7 (e.g., the dashed vertical line) with alower degree of accuracy and specificity.

In another example, the user may want to select an operating point basedon model tuning for PPV and NPV, with the PPV and NPV of an operatingpoint corresponding to the operating point's degree of precision withregard to the probability of disease within an annotated image. Thus, ifthe third set of metric curves 406 was displayed, the user may select anoperating point based on the tuning metric error of an NPV curve 418 anda PPV curve 420. In the depicted example, the model currently utilizesan operating point with a threshold of 0.7 (e.g., the dashed verticalline) which generates a tuning metric error of about 0.5 which maycorrelate to about 50% of the model's output matching the tuningdatasets annotations of positive findings. Based on model tuning, theuser may opt to select a new operating point with a higher or lowerdegree of tuning metric error as best suited to the user's clinicalneeds. For example, the user may select an operating point with a higherPPV value, such as a threshold of 0.9 as indicated by the X. Similarly,the user may want to select an operating point based on a falsepositives per day curve 422, a true positives per day curve 424, and/ora false negatives per day curve based on clinical needs as shown in thefourth set of metric curves 408. After tuning, the user may opt toutilize the model's current operating point (e.g., the dashed verticalline) in which the TPR, FPR, and false negative rate (FNR) are roughlythe same, all occurring less than 12 times within a day. Thus, users mayoptimize the model's performance based on one or more calculated metricsusing the method described herein.

FIGS. 5A and 5B show a non-limiting example of a GUI 500 that medicalstaff may use to tune a static model in a clinical setting according tothe method disclosed herein. GUI 500 is comprised of several drop-downmenus including an Annotated Image Set menu 502, a Tuning Metric menu504, and a Result Output menu 506. In other embodiments, GUI 500 mayhave more or less drop-down menus. For example, a Result Output menu maynot be included and the optimal operating point may be automaticallydetermined and applied after tuning using method 200. Once a user hasselected various options from the drop-down menus of GUI 500, theirselections are displayed under a User Selections banner 508.

To choose a specific annotated tuning dataset, the user may select anannotated image set menu to view a list of body parts/sections. The usermay select the body part/section that corresponds to the area that theuser would like to image. Once a body part/section has been selected, asecond drop-down list may be viewed comprised of anatomical features(e.g., organs and bones) in that body part/section that may be assessedby radiological imaging. Selection of a specific anatomical feature maygenerate a third drop-down menu comprised of different diseases orradiological findings that may be identified and characterized using thestatic model. Once a disease or finding has been selected, a fourthdrop-down list of annotated image sets may be viewed and an image setselected based on user preference to tune the static model using method200. For example, as shown in FIG. 5A, a user of GUI 500 has chosen touse Image set 3 to tune to the AI model before imaging the lungs todetermine if a patient has lung nodules. In another example, anannotated image set menu may be comprised of one drop-down list of imagesets, with each image set labelled according to customer preferences.For instance, image sets may be labelled with disease names and thedegree of the parameter for which the images are annotated (e.g., animage set may be labelled “Lung nodule—high sensitivity,” whereas asecond image set may be labelled “Lung nodule—high specificity”).Alternatively, an annotated image set menu may be comprised of adrop-down list of user names and each user may create sub-lists asdesired (e.g., a first user may choose to organize the image sets asdescribed above, a second user may opt to organize image sets byannotations with a sub-list generated for each disease/finding).

Once an annotated image set has been selected, the user may select atuning metric menu from which a drop-down list of different tuningmetrics may be viewed. For example, as shown in FIG. 5B, a user hasselected to tune the static model for sensitivity using a Tuning Metricmenu 504. Tuning Metric menu 504 only lists sensitivity versusspecificity and PPV & NPV as tuning metric options however, in otherexamples, a tuning metric menu may include additional metrics (e.g.,maximum accuracy, balanced accuracy, false positives, false negatives).In other examples, users may select more than one tuning metric (e.g.,sensitivity versus specificity and PPV and NPV may both be selected orthe menu may include an option to select all metrics). Users may thenuse a result output menu to choose how they would like the results ofthe model tuning displayed. For example, as shown in FIGS. 5A and 5B, auser has used Result Output menu 506 to select that the result output bea graphical representation (e.g., metric curves 400 of FIG. 4). In otherexamples, the result output may be displayed as a summary table or acombination of summary tables and graphs. Once the user has selected anannotated tuning dataset, a tuning metric, and their desired resultoutput, on the user may select a Run Model Tuning button 510 to beginmodel tuning. Once the model has been tuned, a second GUI may be outputcomprised of the result output and allowing for user selection of anoptimal operating point for subsequent image analysis as shown in FIG.6.

FIG. 6 shows a non-limiting example of a GUI 600 that may be outputfollowing model tuning. For example, the user of GUI 500 selected tohave the result output as a graphical representation as shown by themetric curves on the right hand side of GUI 600. GUI 600 furtherincludes a banner stating what disease/finding the model was tuned forin the upper left corner (e.g., lung nodule), under which the value ofthe current selected threshold/operating point is displayed (e.g.,0.70). GUI 600 is further comprised of a Graphical Layout menu 602 thatlists the different graphical output that may be displayed to the user.For example, the user of GUI 500 may have selected to run model tuningusing both sensitivity versus specificity and PPV and NPV as tuningmetrics, with the result output as graphical representations. Thus, theuser of GUI 600 may view graphical data output for sensitivity versusspecificity and/or PPV and NPV by selecting an option listed under theGraphical Layout menu 602. The Graphical Layout menu 602 only listssensitivity versus specificity, PPV and NPV, and all graphs as graphicaloutput options that may be displayed; however, in other examples, agraphical layout menu may include additional metrics (e.g., maximumaccuracy, balanced accuracy, false positives, false negatives).

Further, GUI 600 may include an auto-select menu which may allow theuser to have a default operating point automatically selected based onuser specified criteria. For example, an Auto-select menu 604 mayinclude the maximum Youden index, maximum accuracy, and maximum balancedaccuracy. Thus, if the user selects the maximum Youden index as thecriteria by which an operating point may be automatically selected, theoperating point with the highest Youden index will be determined andapplied for subsequent image analysis. Alternatively, the user may use athreshold selection menu to determine an operating point based on theresult output presented in GUI 600. For example, the user may selectThreshold 3 from a Threshold Selection menu 606 after determiningThreshold 3 as the operating point best suited for image analysis basedon the graphical data presented. Once users have selected whichthreshold or operating point they would like to use based on the resultdata provided (e.g., auto-selected based on selected criteria orspecifically selected), they may select an Apply Selected Thresholdbutton 608 to use said operating point for image analysis.

FIGS. 7A and 7B show a non-limiting example of an annotated clinicalimage 700 that may be part of a tuning dataset used to tune a staticmodel via method 200. An x-ray image 702 may come from an institution'sclinical practice data (e.g., obtained by an x-ray machine at theinstitution) and be annotated by a user (e.g., a radiologist at theinstitution) via input to text box 704 (e.g., directly typing in theannotation). For example, in FIG. 7A, a user has annotated x-ray image702 as positive for a lung nodule as shown by annotation label 706.Alternatively, an x-ray image may be annotated using image segmentation.As shown in FIG. 7B, a user has segmented a region of interest (ROI)within x-ray image 702 that corresponds to a positive finding for a lungnodule. Images annotated via segmentation may be used as a tuningdataset to tune a static model via method 200. The operating points ofthe model may be classified by different thresholds for the transitionpoint between the highest pixel intensity and the lowest pixel intensityforming an edge (e.g., one operating point may determine segmentation onthe midpoint of the edge transition, a second operating point maydetermine segmentation three-fourths of the way through the edgetransition, etc.). For example, annotation by segmentation may output animage mask. Each pixel within the image mask may be labelled with avalue between 0 and 1, with the value representing the probability thepixel shows disease. The image masks output from the annotated data setmay then be used to tune a static model, with a desired operating pointdetermined/selected based on similarity metrics between the image masksand the model output.

In this way, an operating point of a static model may be tuned to enableoptimal desired performance according to the method described herein.The embodiments disclosed herein provide a method that may be employedby customers to easily identify which operating point of a static modelmay best suit their needs based on tuning the model using their ownclinical practice data. The technical effect of tuning an operatingpoint of a static model is that performance of the model may becustomized to best meet the needs of an institution without retrainingthe model or potentially triggering a new regulatory clearance.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralof said elements or steps, unless such exclusion is explicitly stated.Furthermore, references to “one embodiment” of the present disclosureare not intended to be interpreted as excluding the existence ofadditional embodiments that also incorporate the recited features.Moreover, unless explicitly stated to the contrary, embodiments“comprising,” “including,” or “having” an element or a plurality ofelements having a particular property may include additional suchelements not having that property. The terms “including” and “in which”are used as the plain-language equivalents of the respective terms“comprising” and “wherein.” Moreover, the terms “first,” “second,” and“third,” etc. are used merely as labels, and are not intended to imposenumerical requirements or a particular positional order on theirobjects.

This written description uses examples to disclose the invention,including the best mode, and also to enable a person of ordinary skillin the relevant art to practice the invention, including making andusing any devices or systems and performing any incorporated methods.The patentable scope of the invention is defined by the claims, and mayinclude other examples that occur to those of ordinary skill in the art.Such other examples are intended to be within the scope of the claims ifthey have structural elements that do not differ from the literallanguage of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

1. A method, comprising: responsive to a request to tune a static model,obtaining a tuning dataset including a set of medical images; executingthe static model using the set of medical images as input to generatemodel tuning output; determining, for each operating point of a set ofoperating points, a set of tuning metric values based on the tuningdataset and the model tuning output relative to each operating point;selecting an operating point from the set of operating points based oneach set of tuning metric values; and upon a request to analyze asubsequent medical image, outputting a representation of a findingoutput of the static model executed at the selected operating point. 2.The method of claim 1, wherein obtaining the tuning dataset includingthe set of medical images comprises obtaining the set of medical imagesand a corresponding set of labels, each medical image of the set ofmedical images associated with a respective label.
 3. The method ofclaim 2, wherein each label indicates a presence or absence of aspecific finding and/or a region of interest in the associated medicalimage.
 4. The method of claim 3, wherein the set of operating pointsincludes a set of thresholds, each set of tuning metric valuesdetermined by comparing the model tuning output to a different thresholdof the set of thresholds.
 5. The method of claim 4, wherein determiningthe set of tuning metric values for each operating point based on thetuning dataset and the model tuning output relative to each operatingpoint comprises, for a first operating point of the set of operatingpoints, determining a respective first tuning metric value for eachmedical image based on whether an indication of a presence or absence ofa specific finding determined based on the model tuning output for thatmedical image relative to the first operating point matches anindication of a presence or absence of the specific finding of the labelcorresponding to that medical image.
 6. The method of claim 5, whereinselecting the operating point from the set of operating points based onthe set of tuning metric values comprises, for each set of tuning metricvalues, summing the tuning metric values in that set to generate asummary score for each operating point, and selecting the operatingpoint having a summary score that is the closest to a target score. 7.The method of claim 6, wherein outputting the representation of thefinding output of the static model executed at the selected operatingpoint comprises outputting a notification for display on a displaydevice indicating a presence or absence of the specific finding in thesubsequent medical image.
 8. The method of claim 1, wherein outputtingthe representation of the finding output of the static model executed atthe selected operating point comprises outputting a notification fordisplay on a display device indicating whether or not a patient imagedin the subsequent medical image is positioned at a target position.
 9. Amethod, comprising: tuning an operating point of a static modelconfigured to output a presence or absence of a specific finding in oneor more medical images by executing the static model on a set ofannotated medical images and comparing output from the static model foreach annotated medical image of the set of annotated medical images toeach of a plurality of possible operating points, and selecting theoperating point from the plurality of possible operating points thatresults in a target tuning metric; and executing the static model on asubsequent medical image to determine a presence or absence of thespecific finding in the subsequent medical image by comparing output ofthe static model for the subsequent medical image to the selectedoperating point.
 10. The method of claim 9, wherein the target tuningmetric comprises a target sensitivity, a target specificity, a targetaccuracy, a target positive predictive value, and/or a target negativepredictive value.
 11. The method of claim 9, wherein the target tuningmetric comprises maximum accuracy and wherein selecting the operatingpoint from the plurality of possible operating points that results inmaximum accuracy comprises: comparing the output from the static modelfor each annotated medical image of the set of annotated medical imagesto a first possible operating point to determine, for each annotatedmedical image, whether that annotated medical image is positive ornegative for the specific finding; assigning a first tuning metric valueto each annotated medical image based on whether the determination ofthe positive or negative for the specific finding for each annotatedmedical image matches an indication of whether that annotated medicalimage is positive or negative for the specific finding as conveyed by anannotation of that annotated medical image; summing each tuning metricvalue to determine a summary score for the first operating point;determining a summary score for each additional possible operating pointby comparing the output from the static model for each annotated medicalimage to each additional possible operating point and assigning arespective second tuning metric value to each annotated medical imagefor each additional possible operating point; and selecting the possibleoperating point that has the lowest summary score.
 12. The method ofclaim 9, wherein selecting the operating point from the plurality ofpossible operating points that results in the target tuning metriccomprises: determining a specificity value and a sensitivity value foreach possible operating point based on the output from the static modelfor each annotated medical image relative to each of a plurality ofpossible operating points and further based on, for each annotatedmedical image, whether that annotated medical image is positive ornegative for the specific finding as conveyed by an annotation of thatannotated medical image; plotting each specificity value as a functionof a corresponding sensitivity value to form a metric curve; outputtingthe metric curve for display on a display device; receiving a user inputselecting a point on the metric curve; and setting the selectedoperating point as the operating point corresponding to the selectedpoint on the metric curve.
 13. The method of claim 9, wherein the staticmodel is configured to output a presence or absence of a specificdiagnostic finding in one or more x-ray images, wherein the set ofannotated medical images comprises a set of annotated x-ray images, eachx-ray image of the set of annotated x-ray image including an annotationfrom an expert indicating a presence or absence of the specificdiagnostic finding in that x-ray image.
 14. A system, comprising: adisplay device; and non-transitory memory storing instructionsexecutable by a processor to: execute a static model using a set ofmedical images of a tuning dataset as input to generate model tuningoutput; determine, for each operating point of a set of operatingpoints, a set of tuning metric values based on the tuning dataset andthe model tuning output relative to each operating point; select anoperating point from the set of operating points based on each set oftuning metric values; and upon a request to analyze a subsequent medicalimage, display on the display device a finding of the subsequent medicalimage, the finding based on an output of the static model relative tothe selected operating point.
 15. The system of claim 14, wherein thememory and processor are included in an x-ray imaging device, andwherein the subsequent medical image is acquired by the x-ray imagingdevice.
 16. The system of claim 15, wherein at least some images of theset of medical images are acquired with the x-ray imaging device. 17.The system of claim 14, wherein the memory and the processor areincluded in a computing device operably coupled to an imaging device,and wherein the subsequent medical image is acquired with the imagingdevice.
 18. The system of claim 14, wherein selecting the operatingpoint from the set of operating points based on each set of tuningmetric values comprises selecting the operating point from the set ofoperating points based on each set of tuning metric values and furtherbased on a target tuning metric.
 19. The system of claim 18, wherein thetarget turning metric comprises a target sensitivity, a targetspecificity, a target accuracy, a target positive predictive value,and/or a target negative predictive value.
 20. The system of claim 14,wherein the finding of the subsequent medical image is determined bycomparing the output of the static model to the selected operating pointand selecting the finding based on whether the output of the staticmodel is greater than the selected operating point.